ASTROIDE: A Unified Astronomical Big Data Processing Engine over Spark
نویسندگان
چکیده
منابع مشابه
Architecture of processing and analysis system for big astronomical data
This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astr...
متن کاملBig Data: Astronomical or Genomical?
Genomics is a Big Data science and is going to get much bigger, very soon, but it is not known whether the needs of genomics will exceed other Big Data domains. Projecting to the year 2025, we compared genomics with three other major generators of Big Data: astronomy, YouTube, and Twitter. Our estimates show that genomics is a "four-headed beast"--it is either on par with or the most demanding ...
متن کاملConquering Big Data with Spark
Today, big and small organizations alike collect huge amounts of data, and they do so with one goal in mind: extract "value" through sophisticated exploratory analysis, and use it as the basis to make decisions as varied as personalized treatment and ad targeting. To address this challenge, we have developed Berkeley Data Analytics Stack (BDAS), an open source data analytics stack for big data ...
متن کاملRealtime, Distributed Big Data Indexing System using Spark for Term-Based Search Engine on HPC Clusters
Realtime Indexing has recently become an active research area due to the size and growth rate of today’s internet content. It is crucial for search engine to not only be able to index large datasets, but also index at fast rate. In this paper, we propose a approach to index content at real time when a file is being newly created. In this paper, we first describe in detail our implementation wit...
متن کاملSpark-BDD: Debugging Big Data Applications
Apache Spark has become a key platform for Big Data Analytics, yet it lacks complete support for debugging analytics programs. As a result, the development of a new analytical toolkit can be a painstakingly long process [7, 2, 4]. To fill this gap, we are developing Spark-BDD (Big Data Debugger), which brings a traditional interactive debugger experience to the Spark platform. Analytic programm...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: IEEE Transactions on Big Data
سال: 2020
ISSN: 2332-7790,2372-2096
DOI: 10.1109/tbdata.2018.2873749